Pulse width modulation algorithm

ABSTRACT

In display systems employing spatial light modulators, the OFF-state light from OFF-state pixels of the spatial light modulator can be captured and directed back to the pixels of the spatial light modulator so as to recycle the OFF-state light in the display system. Bitplanes derived from the desired image to be produced are calibrated to include the recycled off-state light to properly produce the desired image using the display system.

This application is a divisional of application Ser. No. 11/782,897,filed Jul. 25, 2007 which claims the benefit of provisional ApplicationNo. 60/909,877, filed Apr. 3, 2007 and is a continuation-in-part ofapplication Ser. No. 11/696,033, filed Apr. 3, 2007, now U.S. Pat. No.7,876,340.

CROSS REFERENCE TO RELATED CASES

This US patent application claims priority under 35 U.S.C. 119(e) ofprovisional US patent application “A PULSE WIDTH MODULATION ALGORITHM”to Russell, Ser. No. 60/909,877 filed Apr. 3, 2007; and this US patentapplication is a continuation-in-part of co-pending US patentapplication “A PULSE WIDTH MODULATION ALGORITHM” to Russell, Ser. No.11/696,033, filed Apr. 3, 2007. The subject matter of each isincorporated herein by reference in its entirety.

This US patent application is also related to US patent application“OFF-STATE LIGHT RECAPTURING IN DISPLAY SYSTEMS EMPLOYING SPATIAL LIGHTMODULATORS” to Russell, Ser. No. 11/696,044 filed Apr. 3, 2007, thesubject matter of which is incorporated herein by reference in itsentirety.

TECHNICAL FIELD

The technical field of the examples to be disclosed in the followingsections relates to the art of display systems, and more particularly,to pulse-width-modulation techniques for use in display systemsemploying spatial light modulators.

BACKGROUND

In current imaging systems that employ spatial light modulators composedof individually addressable pixels, a beam of incident light is directedto the pixels of the spatial light modulator. By setting the pixels atan ON state, the incident light is modulated onto a screen so as togenerate bright image pixels on the screen, wherein such modulated lightis referred to as the ON-state light; and the pixels at the ON state arereferred to as ON-state pixels. By setting the pixels at an OFF state,the incident light is modulated away from the screen so as to cause darkpixels on the screen, wherein such modulated light is referred to asOFF-state light; and the pixels at the OFF state are referred to asOFF-state pixels. For obtaining a high contrast ratio, the OFF-statelight is dumped or discarded by the imaging systems, which on the otherhand, reduces the optical efficiency of the imaging system.

SUMMARY

In one example, a method for displaying an image is disclosed herein.The method comprises: directing a beam of incident light onto an arrayof pixels of a spatial light modulator, wherein each pixel is capable ofbeing operated at a first state and a second state; modulating the beamof incident light into a first portion of light by pixels at the firststate and a second portion of light by the pixels at the second statebased on a set of bitplanes, further comprising: displaying eachbitplane by the pixels for a time period that is determined based on anumber of pixels in said each bitplane that cause the pixels of thespatial light modulator to the second state; directing the first portionof light from the spatial light modulator onto a display target, and thesecond portion of light from the spatial light modulator away from thedisplay target; and recycling the second portion of light back to thepixels of the spatial light modulator.

In another example, a device for use in a display system employing aspatial light modulator having an array of pixels with each pixelcapable of being operated at a first state and a second state that isdifferent from the first state is disclosed herein. The devicecomprises: first means for deriving the bitplanes based on a sequence ofcolor duty cycles; and second means for determining a clock speed atwhich the spatial light modulator is to be operated in displaying thebitplanes based on a number of off-states in each bitplane.

In yet another example, a display system is disclosed herein. The systemcomprises: a light source capable of providing a light beam; a spatiallight modulator having an array of individually addressable pixels eachbeing capable of modulating the light beam into a first portion of lightwhen said each pixel is at a first state and a second portion of lightwhen said each pixel is at a second state based on a set of bitplanes; alight recycling mechanism capable of recycling the second portion oflight back to the spatial light modulator; and a system controller forcontrolling an operation of the spatial light modulator, furthercomprising a device that comprises: a bitplane module for deriving thebitplanes based on a sequence of color duty cycles; and a clock speedcalculator in connection to the bitplane module for determining a clockspeed at which the spatial light modulator is to be operated indisplaying the bitplanes based on a determination of an amount of lightto be recycled by the light recycling mechanism in displaying at leastone of the derived bitplanes.

In still yet another example, a method for displaying a color imagecomposed of a set of color image components is disclosed herein. Themethod comprises: directing a sequence of light beams of differentcolors to an array of pixels of a spatial light modulator; at a timewhen one of the light beams of a color is illuminating the spatial lightmodulator, modulating said one of the light beams into a first portionof modulated light and a second portion of modulated light by the pixelsof the spatial light modulator; directing the first portion of modulatedlight onto a display target; and recycling at least a part of the secondportion of the modulated light back to the pixels of the spatial lightmodulator.

In yet another example, a method for displaying an image is disclosedherein. The method comprises: directing a beam of light on an array ofpixels of a spatial light modulator; modulating said beam of light bythe pixels of the spatial light modulator into modulated light based ona modified bitplane; recycling at least a part of the modulated lightback to the pixels of the spatial light modulator; and wherein saidmodified bitplane is obtained by a process comprising: deriving abitplane from the image; and modifying the derived bitplane such that afirst pixel of the derived bitplane is modified in a differently from asecond pixel of the derived image.

In yet another example, a method of displaying a color image frame isdisclosed herein. The method comprises: defining a set of colorscomprising first and second colors for representing the color imageframe; obtaining an energy distribution of the color image frame in theset of colors; re-distributing the energy in the set of colors by movingthe energy of the first color into the energy of the second color; anddisplaying the image frame based on the re-distributed energy in set ofcolors.

In yet another example, a display system is disclosed. The systemcomprises: a light source capable of providing a light beam; a spatiallight modulator having an array of individually addressable pixels eachbeing capable of modulating the light beam into a first portion of lightwhen said each pixel is at a first state and a second portion of lightwhen said each pixel is at a second state based on a set of bitplanes; alight recycling mechanism capable of recycling the second portion oflight back to the spatial light modulator; and a system controller forcontrolling an operation of the spatial light modulator, furthercomprising: a data formatter capable of converting the pixel data of theinput image into the set of bitplanes; an average pixel calculatorhaving an input connected to the input image for calculating an averagepixel value of the input image; a real-time bit counter having an inputcoupled to the bitplanes for calculating a number of pixels in eachbitplane that cause the pixels of the spatial light modulator to be atthe second state; a clock speed calculator having a set of inputsconnected to an output of the average pixel value calculator and anoutput of the real-time bit counter for calculating a clock speed basedon the calculated average pixel data value and said number of pixels ineach said bitplane; a clock speed adjustor having an input connected toan output of the clock speed calculator and an input coupled to asequence of clock signals of the spatial light modulator provided by aclock of the display system; and a controller having an input connectedto an output of the clock speed adjustor for operating the spatial lightmodulator at the adjusted clock speed.

In yet another example, a method for displaying a sequence of imageframes using an array of individually addressable pixels of a spatiallight modulator is disclosed. The method comprises: deriving a set ofbitplanes from each image frame, wherein each bitplane has an indexrepresenting a relative position of the bitplane in the set ofbitplanes; modulating, by the pixels of the spatial light modulator, abeam of incident light into ON-state and OFF-state light based on thebitplanes, further comprising: displaying a first bitplane of a firstset of bitplanes derived from a first image frame of the image sequencethat is displayed by the pixels of the spatial light modulator for afirst time period; displaying a second bitplane of a second set ofbitplanes derived from a second image frame of the image sequence thatis displayed by the pixels of the spatial light modulator for a secondtime period; and wherein the first and second bitplanes have the sameindex; and the first and second time periods are different; directingthe OFF-state light away from a display target and the ON-state lightonto the display target; and re-routing the OFF-state light back to thepixels of the spatial light modulator.

In yet another example, a method for displaying an image is disclosedherein. The method comprises: directing a beam of light on an array ofpixels of a spatial light modulator; modulating said beam of light bythe pixels of the spatial light modulator into modulated light based ona modified bitplane; recycling at least a part of the modulated lightback to the pixels of the spatial light modulator; and wherein saidmodified bitplane is obtained by a process comprising: deriving abitplane from the image; and modifying the derived bitplane such that afirst pixel of the derived bitplane is modified in a differently from asecond pixel of the derived image.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 diagrammatically illustrates a diagram of an exemplary displaycomprising an off-state light recycling mechanism;

FIG. 2 is a block diagram illustrating an exemplary off-state lightrecycling mechanism illustrated in FIG. 1;

FIG. 3 shows a diagram of the maximum gain vs. the recycling efficiency;

FIG. 4 shows a diagram of the gain vs. the average picture-level (APL)for different recycling efficiencies;

FIG. 5 is a flow chart of an exemplary pulse-width-modulation algorithmfor use in determining bitplane data for a spatial light modulator in adisplay system that employs an off-state light recycling mechanism asillustrated in FIG. 1;

FIG. 6 is a flow chart showing the steps executed for calculating theframe gain due to the off-state recycling in the pulse-width-modulationalgorithm shown in FIG. 5;

FIG. 7 is a flow chart showing the steps executed for calibrating theset of bitplanes to present the calibrated pixel data in thepulse-width-modulation algorithm shown in FIG. 5;

FIG. 8 shows a diagram of the calibrated pixel data sorted left to rightfrom smallest to largest for the red color component of the image shownin FIG. 21 b;

FIG. 9 and FIG. 10 show an exemplary method of calibrating the firstbitplane for the red color image component of the image in FIG. 21 b;

FIG. 11 and FIG. 12 show a method of calibrating the second bitplane forthe red color image component of the image in FIG. 21 b;

FIG. 13 through FIG. 18 show a method of consecutively calibrating theremaining bitplanes for the red color image component of the image inFIG. 21 b;

FIG. 19 and FIG. 20 show an exemplary method of presenting residualenergies in the pixel data;

FIG. 21 a and FIG. 21 b show exemplary images with and without thebrightness boost, wherein the image having the brightness boost ispresented by the pulse-width-modulation algorithm as illustrated in theflow chart of FIG. 5;

FIG. 22 schematically illustrates a method for adjusting a sequence ofcolor duty cycles so as to display bitplanes at adjusted clock speeds inthe presence of off-state light recycling;

FIG. 23 demonstrates an exemplary system and method for displayingbitplanes with dynamically adjusted clock speed in the presence ofoff-state light recycling;

FIG. 24 and FIG. 25 schematically illustrates a method for calculatingoff-state pixels in bitplanes that are to be displayed by the spatiallight modulator whose pixels are divided into reset groups; wherein FIG.24 illustrates the array of pixels of the spatial light modulatordivided into reset groups; and FIG. 25 illustrates an exemplarybit-counter for calculating off-state pixels based on reset groups;

FIG. 26 is a flow chart having steps executed in displaying bitplaneswith adjusted clock speed in the presence of off-state light recycling;

FIG. 27 schematically illustrates an exemplary sequence of color dutycycles;

FIG. 28 a through FIG. 28 c illustrate an exemplary method for adjustingthe sequence of color duty cycles in FIG. 27 so as to display bitplanesat adjusted clock speeds in the presence of off-state light recycling;and

FIG. 29 a and FIG. 29 b schematically illustrate an exemplary method ofre-distributing energies in the color image components associated withthe sequence of color duty cycles in FIG. 27.

DETAILED DESCRIPTION OF SELECTED EXAMPLES

In a typical existing display system employing a spatial lightmodulator, ON-state light from ON-state pixels of the spatial lightmodulator propagates towards the screen of the display system so as togenerate a bright image pixel on the screen. The off-state light fromoff-state pixels of the spatial light modulator travels away from thescreen so as to result in a dark pixel on the screen. Such off-statelight is often dumped or discarded by the display system, which reducesthe optical efficiency of the display system.

As an aspect of this disclosure, a mechanism for recycling the off-statelight is provided, as will be discussed in the first part of thisdisclosure. Because most current display systems employing spatial lightmodulators operate based on bitplanes; and the bitplanes may causedifferent number of pixels at the OFF-state, the intensity of therecycled off-state light varies over time or over bitplanes. Suchvariation, in turn, causes distortion of the displayed image. Thisproblem can be solved by calibrating the set of bitplanes derived fromthe desired image to include the energy of the recycled off-state light,as will be discussed in the second part of this disclosure. This problemcan alternatively be solved by displaying the bitplanes derived from thedesired image at dynamically adjusted clock speed, as will be discussedin the third part of this disclosure.

Off-State Light Recycling Mechanism

The off-state light from off-state pixels at a time can be captured andrerouted back to the pixels of the spatial light modulator. The reroutedoff-state light, when illuminating the on-state pixels of the spatiallight modulator, is converted to the ON-state light that can beprojected to the screen so as to increase the brightness of theprojected image. When illuminating the off-state pixels, the reroutedoff-state light will be re-captured and then rerouted again to thepixels of spatial light modulator. This off-state light capturing andrerouting process is referred to as “off-state light recycling.” Adevice having the capability of off-state light recycling in displaysystems is referred to as an off-state light recycling mechanism.

As an example, FIG. 1 diagrammatically illustrates an exemplary displaysystem in which an off-state recycling mechanism is implemented. In thisexample, display system 100 comprises light source 102, off-state lightrecycling mechanism 104, spatial light modulator 108, projection lens110, display target 112, and system controller 124 that furthercomprises data processing unit 126. Multimedia source 122, such as videoand image sources, is connected to the system controller for providingmultimedia signals. It is noted that the multimedia source may or maynot be a member of the display system. The display target (112) can be ascreen on a wall or the like, or can be a member of a rear projectionsystem, such as a rear projection television. In fact, the displaysystem can be any suitable display system, such as a front projector, arear projection television, or a display unit for use in other systems,such as mobile telephones, personal data assistants (PDAs), hand-held orportable computers, camcorders, video game consoles, and other imagedisplaying devices, such as electronic billboards and aestheticstructures.

Light source 102 provides light for the imaging system. The light sourcemay comprise a wide range of light emitting devices, such as lasers,light-emitting-diodes, arc lamps, devices employing free space orwaveguide-confined nonlinear optical conversion and many other lightemitting devices. In particular, the light source can be a light sourcewith low etendue, such as solid state light emitting devices (e.g.lasers and light-emitting-diodes (LEDs)). When solid-state lightemitting devices are used, the light source may comprise an array ofsolid-state light emitting devices capable of emitting different colors,such as colors selected from red, green, blue, and white. Because asingle solid-state light emitting device generally has a narrowcharacteristic bandwidth that may not be optimal for use in displaysystems employing spatial light modulators, multiple solid-state lightemitting devices can be used for providing light of each color so as toachieve optimal bandwidth for specific display systems. For example,multiple lasers or LEDs with slightly different characteristic spectra,such as 20 nm or less characteristic wavelength separation, can be usedto produce a color light such that the characteristic spectra of themultiple lasers or LEDs together form an optimal spectrum profile of thedisplay system. Exemplary laser sources are vertical cavity surfaceemitting lasers (VCSEL) and Novalux™ extended cavity surface emittinglasers (NECSEL), or any other suitable laser emitting devices.

Spatial light modulator 108 comprises an array of individuallyaddressable pixels for spatially modulating the incident light onto oraway from projection lens 110 that projects the modulated light ontoscreen 112 so as to reproduce images. The spatial light modulator maycomprise pixels of many different natures, such as reflective anddeflectable micromirrors and liquid-crystal-on-silicon (LCOS) devices.The pixels can be operated using binary or non-binary modes. In thebinary mode, each pixel is switched between an ON and OFF state. At theON state, each pixel modulates the incident light onto the projectionlens (110). At the OFF state, each pixel modulates the incident lightaway from the projection lens. The ON-state light arrives at the screen(112) so as to construct the desired image; and the OFF-state isrecycled by off-state light recycling mechanism 104 and redirected tothe spatial light modulator, which will be discussed afterwards. Thepixels of the spatial light modulator alternatively can be operated at anon-binary mode, such as an analog mode wherein multiple intermediatestates are defined between an ON and OFF state; and the intermediatestates may or may not be continuous between the ON and OFF states. Ineither binary or non-binary operation mode, color and gray images can beproduced using a pulse-width-modulation technique, examples of whichwill be discussed afterwards.

OFF-state light recycling mechanism 104 is optically coupled to thepropagation path of the off-state light that is modulated from thepixels of the spatial light modulator (108) such that the off-statelight from the pixels at the OFF state of the spatial light modulatorcan be recaptured by the off-state light recycling mechanism. Forredirecting the recaptured off-state light back to the pixels of thespatial light modulator, the OFF-state light recycling mechanism has alight exit end that is aligned to the propagation path of the incidentlight to the pixels of the spatial light modulator.

In the example illustrated in FIG. 1, incident light 106 from the lightsource is directed to spatial light modulator 108 that modulates theincident light (116) into ON-state light 107 and OFF-state light 114.The ON-state light travels towards projection lens 110; and is projectedonto screen 112 by projection lens 110. OFF-state light 114 isrecaptured by OFF-state light recycling mechanism 104 that is capable ofconverting the recaptured OFF-state light into incident light 116 andredirecting light 116 to illuminate pixels of spatial light modulator108. At the spatial light modulator, incident light 116 may be modulatedinto ON-state light 117 that is collected by projection lens 110 and theOFF-state light that is recaptured by the off-state light recyclingmechanism (104).

Because the OFF-state light from the spatial light modulator can berecaptured and redirected to the spatial light modulator, this recyclingprocess improves the brightness of images produced on the screen. Suchbrightness improvement can be mathematically described as brightnessgain as expressed in equation 1:

$\begin{matrix}{I = {{I_{o}G} = {I_{o}\frac{1}{1 - {ɛ\left( {1 - x} \right)}}}}} & \left( {{Eq}.\mspace{14mu} 1} \right)\end{matrix}$In equation 1, G is the brightness gain due to OFF-state lightrecycling; I is the illumination intensity of light arriving at thescreen including the recycled OFF-state light; and I_(o) is theillumination intensity of light arriving at the screen without OFF-satelight recycling. ε is the OFF-state light recycling efficiency that isdefined as the fraction of the OFF-state light that re-illuminates thepixels of the spatial light modulator after a recycling process,compared to the total amount of OFF-state light to be recycled by therecycling process. x is the normalized number of ON-state pixels of thespatial light modulator at a time (e.g. during a bitplane time).Specifically, x can be expressed as equation 2:

$\begin{matrix}{x = \frac{N_{ON}}{N_{total}}} & \left( {{Eq}.\mspace{14mu} 2} \right)\end{matrix}$wherein N_(ON) is the number of ON-state pixels at a time; and N_(total)is the total number of pixels involved in modulating the incident light.It is noted that N_(total) may or may not be the total number of pixelsof the spatial light modulator, especially when the spatial lightmodulator comprises active and inactive pixel areas. Pixels in inactivepixel areas of spatial light modulators are those pixels whose states inimage display operations are independent from the data (e.g. bitplanedata) derived from desired images; whereas pixels in active pixel areasare those whose states are associated with or determined by the imagedata.

Recycling efficiency ε is primarily determined by the optical design ofthe off-state light recycling mechanism and the optical coupling of theoff-state light recycling mechanism to the display system, particularlyto the propagation path of the OFF-state light from the spatial lightmodulator and the propagation path of the light incident to the spatiallight modulator. Ideally, ε is 100%. In practice, ε may be less than100% due to imperfect optical coupling of the off-state light recyclingmechanism to the propagation path of the off-state light from thespatial light modulator and/or to the propagation path of the incidentlight to the spatial light modulator and/or due to light leakage fromimperfect optical design of the off-state light recycling mechanism. Tomaximize the brightness gain, it is preferred that ε is maximized. Inother examples, however, maximizing off-state light recycling may beimpeded by other preferred system features, which results in balancebetween off-state recycling and the preferred features. For example, theoff-state light recycling mechanism and/or the system design is desiredto be cost-effective or desired to be volume compact or other reasons,poor ε may be selected. In any instances, it is preferred that ε is 10%or more, such as 20% or more, 30% or more, 40% or more, 50% or more, 60%or more, and 70% or more. As an example, table 1 shows the brightnessgain achieved from different fractions of ON-state pixels (which can beconverted to the number of ON-state pixels using equation 2) by assumingthat the recycling efficiency ε is 60%.

TABLE 1 % of ON-state pixels 0 10 20 30 40 50 60 70 80 90 100 Brightness2.5 2.17 1.92 1.72 1.56 1.43 1.32 1.22 1.14 1.06 1 gain

An exemplary variation of the maximum gain with the recycling efficiencyis presented in FIG. 3. The diagram in FIG. 3 assumes that all pixels ofthe spatial light modulator are at the OFF state. Accordingly, equation1 is reduced to equation 3 with the recycling efficiency being thevariable as shown in the following:

$\begin{matrix}{G = \frac{1}{1 - ɛ}} & \left( {{Eq}.\mspace{14mu} 3} \right)\end{matrix}$As can be seen in FIG. 3, the maximum gain is 1 when the recyclingefficiency ε is 0; and the maximum gain is 5 when ε is 0.8.

Because the gain is due to the off-state recycling, the amount of gainobtained through off-state recycling depends on the number of off-statepixels of the spatial light modulator during the recycling process. Asan example, FIG. 4 presents a diagram of the gain vs. theaverage-picture-level (APL) in a bitplane with different curvesrepresenting different recycling efficiencies. The APL is defined as thefraction of the ON-state pixel data (e.g. the total number of “1”) in abitplane. As can be seen in FIG. 4, gain increases as APL decreases. Asubstantially white image has least gain, and thus least brightnessboost; whereas a substantially dark image has the most gain, and thusthe most brightness boost.

In addition to the brightness improvement as discussed above, theoff-state light recycling has many other benefits. For example, theoff-state recycling can also be used to increase the lifetime of thelight source of the imaging system and/or to reduce the powerconsumption of the imaging system. Specifically, the light source can beoperated as a lower power, as compared to imaging operations withoutoff-state light recycling, during imaging operations but withoutsacrificing the brightness of the reproduced images. Operating the lightsource at reduced power certainly helps to increase lifetime of thelight source, especially solid-state light sources, such as lasers andLEDs. Moreover, reduced power also reduces heat generated by the lightsource, which in turn increases lifetime of other components in thesystem by for example, reducing the commonly existing aging effect.

The off-state light recycling mechanism (104) as illustrated in FIG. 1can be implemented in many possible ways, one of which is schematicallyillustrated in FIG. 2. Referring to FIG. 2, off-state light recyclingmechanism 104 comprises optical diffuser 130, optical integrator 132,condensing lens 140, and prism assembly 142. For illustrating therelative positions of the off-state light recycling mechanism in theimaging system, spatial light modulator 108 and projection lens 110 inFIG. 1 are also shown in the figure.

Optical diffuser 130 is provided herein for homogenizing the light beamincident thereto and transforming the incident light beam, especiallynarrow-band or narrow-angle light beans from solid-state light emittingdevices, into light beams with pre-determined illumination fieldprofiles. A narrow-angle light beam is referred to a light beam with asolid-angle extension of 5 degrees or less, such as 2 degrees or less, 1degree or less, 0.5 degree or less, and 0.2 degree or less. Thehomogenization capability of the optical diffuser is enabled by randomlyor regularly deployed scattering centers. The scattering centers can belocated within the body of the diffuser or in (or on) a surface(s) ofthe diffuser, which constitute the features responsible for directingthe incident light into various spatial directions within the spread ofthe optical diffuser. Depending upon different locations of thescattering centers, the optical diffuser can be a volume opticaldiffuser where the scattering centers are within the bulk body of thediffuser, or a surface diffuser where the scattering centers are on thesurface of the bulk body of the diffuser. In one example, the opticaldiffuser can be a surface diffuser, such as a standard engineereddiffuser. Even though not required, the optical diffuser can be usedwhen the light source (102 in FIG. 1) employs solid state (or narrowband) light sources. In other examples, such as the light source usesarc lamps, the optical diffuser may be replaced by an optical lens, suchas a condensing lens, which is not shown in the figure. A lens combinedwith smaller angle or spatial diffusers can also be used.

The optical integrator (132) comprises opening 136 formed in end wall134 of the optical integrator. Side wall 134 has interior surface coatedwith a reflective layer for reflecting the light incident thereto. Inparticular, the interior surface of side wall 134 is used to reverse thedirection of the incident light such that the off-state light recapturedat the other end (138) of optical integrator 132 can be bounced back totravel towards the spatial light modulator. For this purpose, thereflective layer coated on the interior surface of side wall 134 can bea totally-internally-reflecting (TIR) surface for the OFF-state light.

Opening 136 provided in side wall 134 is designated for collecting thelight beams from the light source and directing the collected lighttowards the spatial light modulator (108). Accordingly, opening 136 isoptically aligned to the propagation path of the incident light from thelight source, as illustrated in the figure.

Because the opening (136) is provided to collect the incident light andthe opening is in the side wall 134 that is designated to bounce therecaptured off-state light, the opening has a preferred dimension suchthat off-state light leakage from the opening is minimized whilecollection of the incident light from the light source is maximized. Theopening may have a dimension that matched to the dimension of the lightincident thereto, such as the dimension of the illumination field of thelight beam at the location of side wall 134. As an example, the width orheight of the opening can be 1 mm or less, such as 0.5 mm or less, and0.2 mm or less. The opening may have any desired shape, such as circle,rectangle, and square.

The other end (138) of optical integrator 132 is designated to capturethe off-state light from the spatial light modulator (108). To maximizethe capturing of the off-state light, side 138 of optical integrator 132is substantially open; and the opened portion is optically aligned tothe propagation path of the off-state light from the spatial lightmodulator. In particular, the opening portion of side 138 can beoptically aligned to the illumination field of the off-state light atthe location of side 138. Even though it is shown in the figure thatside 138 and side 136 are substantially parallel and substantially havethe same dimension, it is not required. In other examples, side 138 mayhave a shape and/or a dimension different from that of side 134, inwhich instance, optical integrator 132 can be tapered or extended fromone end (e.g. side 134) to the other (e.g. side 138). Alternatively,optical integrator 132 can be assembled with another optical integratoror a suitable optical element (e.g. lens) such that capturing theoff-state light from the spatial light modulator can be maximized.

Optical integrator 132 may have a solid body, such as a body filled withan optical material (e.g. glass) that is transmissive to the incidentlight. The optical integrator may alternatively comprise a hollowedbody, such as an empty space surrounded by multiple reflective walls,one end-side wall 134, and the other end-side wall 138, as discussedabove.

The incident light (106), including the light from the light source andthe recycled light from the off-state light recycling mechanism, is thenguided to the spatial light modulator by condensing lens 140 and prismassembly 142. For properly directing the incident light onto the pixelsof the spatial light modulator (108) and spatially separating theON-state and OFF-state light, the prism assembly employs TIR surface146. Specifically, TIR surface 146 is optically disposed such that theincident light can be reflected to the spatial light modulator at apre-determined direction; the off-state light (114) from the pixels atthe OFF state can be directed towards side 138 of the off-state lightrecycling mechanism; and the ON-state light (109) from the spatial lightmodulator can travel through the TIR surface towards the projection lens(110). These can be achieved by aligning the TIR surface (146) such thatthe incident light and OFF-state light impinge the TIR surface atincident angles equal to or greater than the critical angle of the TIRsurface; whereas the ON-state light impinges the TIR surface at anincident angle less than the critical angle of the TIR surface.

Condensing lens 140 is provided to form a proper illumination field onthe TIR surface (146) such that the image of such illumination fieldprojected on the spatial light modulator by the TIR surface has a properoptical profile. For example, the profile has an illumination areamatching the pixel area of the spatial light modulator and/or theillumination intensity is substantially uniform across the pixel area. Aproper optical profile can be achieved by adjusting the relativepositions of condensing lens 140, TIR surface 146, and spatial lightmodulator 108.

In the example shown in FIG. 2, optical integrator 132 is disposed onthe optical path of the light from the light source. A benefit of thisconfiguration is that the recycled off-state light can be re-directed tothe spatial light modulator along the same propagation path of theincident light from the light source, thus simplifying the opticaldesign. In other alternative examples, the optical integrator can bedisposed such that the optical axis of the optical integrator is notaligned to the incident light path. In this instance, opening 136 maynot be formed. Moreover, alternative to using a prism assembly with aTIR surface as shown in FIG. 2, the off-state recycling mechanism canemploy an optical fiber or other suitable optical devices.

It is noted that FIG. 1 and FIG. 2 illustrate only one of many possibleoff-state light recycling mechanisms and display systems using the same.Other variations are also applicable, such as those set forth in USpatent application “OFF-STATE LIGHT RECAPTURING IN DISPLAY SYSTEMSEMPLOYING SPATIAL LIGHT MODULATORS” to Russell et al., Ser. No.11/696,044 filed on the same say as this US patent application, thesubject matter of which is incorporated herein by reference in itsentirety.

A Pulse-Width-Modulation Algorithm

For properly reproducing grayscale and color images, each pixel of thespatial light modulator (e.g. 108 in FIG. 1) can be turned on and off ata rate faster than the human eyes can perceive such that the pixelappears to have an intermediate intensity proportional to the fractionof the time when the pixel is on. This method is generally referred toas pulse-width-modulation (PWM). Specifically, each image pixel of agrayscale image is represented by a plurality of pixel data bits (e.g.4-bits, 8-bits, 16-bits, 32-bits, and 64-bits); and each pixel data bitis assigned a significance. Each time the pixel of the spatial lightmodulator is addressed, the value of the pixel data bit determineswhether the addressed pixel is on or off; and the bit significancedetermines the duration of the pixel at the on-state or the off-state. Acollection of pixel data bits of the same significance for the imagepixels is referred to as a bitplane. During a frame period, a number ofbitplanes is displayed by the pixels of the spatial light modulator soas to produce grayscale levels of the desired image. It is noted that abitplane can be a collection of pixel data bits for all or a portion ofimage pixels of the desired image. As a way of example as set forth inU.S. Pat. No. 6,201,521 to Doherty issued Mar. 31, 2001, the subjectmatter being incorporated herein by reference in its entirety, the pixelarray of a spatial light modulator can be arranged into multiple resetgroups. A set of groups of bitplanes is derived from an image to bedisplayed (the desired image) such that each bitplane corresponds to areset group that displays said each bitplane; the bitplanes in eachgroup have the same significance; and all bitplanes in each groupcollectively correspond to all pixel data bits having the samesignificance of the desired image.

At a particular time, bitplanes of the same group (having the samesignificance) can be displayed by the pixels of the spatial lightmodulator. Alternatively, bitplanes of the same group can be displayedat different time periods. In other words, the bitplanes being displayedby the pixels of all reset groups of the spatial light modulator at eachtime can be from the same group (of the same significance); or can befrom different groups (of different significances).

Conversion of the pixels data (e.g. RGB data and YPbPr) of the incomingimage or video signals into bitplane data can be performed in many ways,such as those set forth in U.S. Pat. No. 7,075,506 to Morgan, issuedJul. 11, 2006, U.S. Pat. No. 5,663,749 to Farris issued Sep. 2, 1997,and U.S. patent application Ser. No. 10/648,608 to Richards filed Aug.25, 2003, the subject matter of each being incorporated herein byreference in its entirety. Such data conversion can be accomplished by adesignated functional module, such as data processing unit 126, whichcan be a member of system controller 124 that is used to controloperations of other functional members of the display system, as shownin FIG. 1. It is noted that the data processing unit can be a softwaremodule having a set of computer-executable instructions or an electroniccircuit, such as a field-programmable-gate-array circuit,application-specific-integrated-circuit, and other suitable circuits.

Because different bitplanes can cause different numbers of on-state andoff-state pixels in the spatial light modulator, the amount of recycledoff-state light varies over bitplanes for a given off-state lightrecycling mechanism. Such variation in turn causes unwanted illuminationintensity changes on the screen in reproducing the desired image. As aconsequence, the bitplanes derived from the desired image when displayedby the pixels of the spatial light modulator do not result in thedesired image on the screen. This problem can be solved by many methods,one of which is by deriving the bitplanes from the desired image whileincluding the illumination intensity variations due to recycledoff-state light such that the derived bitplanes, when displayed by thepixels of the spatial light modulator in the presence of off-state lightrecycling, can yield the desired image on the screen. This bitplanederivation process can be performed by the data processing unit (126) asshown in FIG. 1.

As an example, FIG. 5 shows a flow chart having steps executed forperforming an exemplary bitplane derivation process. Because thebitplane derivation process is intended to generate bitplanes whileincluding the effect of off-state light recycling, parameterscharacterizing the off-state light recycling process, such as recyclingefficiency ε, as well as other parameters characterizing the incomingvideo and image signals, such as parameters characterizing chromaticand/or intensity components of the incoming video or image signals, areinitialized (step 150). Based on the initialized parameters, frame gaing_(frame) is calculated (step 152) based on gains g_(i) of chromaticcomponents of the incoming image frame due to off-state light recyclingduring the frame period. An exemplary calculation of frame gain is shownin the flow chart in FIG. 6.

Referring to FIG. 6, the gain due to off-state light recycling iscalculated for each color component (step 162) in the linear light spaceafter de-gamma application. For example, assuming the incoming imageframe comprises red, green, and blue color image components, gainsg_(red), g_(green), and g_(blue) for red, green, and blue color imagecomponents are calculated using equation 4 shown in the following:

$\begin{matrix}{{g_{i} = \frac{1}{1 - {ɛ\left( {1 - A_{i}} \right)}}}{{i = {red}},{green},{{and}\mspace{14mu}{blue}\mspace{14mu}{colors}}}} & \left( {{Eq}.\mspace{14mu} 4} \right)\end{matrix}$In equation 4, A_(i) is the average pixel level (the average value ofthe pixel data in linear light space) of the i^(th) color imagecomponent during the frame period. Given the calculated gain for eachcolor components, the frame gain g_(frame) is determined (step 164) suchthat the frame gain g_(frame) is equal to or less than the minimum gainof the color image components, which can be expressed as equation 5:g _(frame)≦Min(g _(i))  (Eq. 5)In one example, frame gain g_(frame) is set to the minimum gain of thecolor image components. In other examples, the frame gain g_(frame) isless than the minimum gain of the color image components by apredetermined value Δ that can be 20% or less, 10% or less, and 5% orless of the minimum gain of the color image components.

Referring again to FIG. 5, with the calculated frame gain at step 152,time periods for displaying bitplanes to be derived in the followingfrom each color image color component of the desired image arecalculated so as to include the effect the off-state light recycling(step 153). As compared to existing bitplane display technologieswherein each bitplane is displayed for a time period that is determinedby the significance or weight of the bitplane, the bitplane in anexample of this disclosure is instead displayed by the pixels of aspatial light modulator for a time period that is related the off-staterecycling efficiency as well as the significance of the bitplane. Thedisplay periods can be calculated in many ways. As an example, assumingthe specific color image component is to be displayed for 1 unit time,this one unit time can be dissected into multiple time slots for thebitplanes, as well as a S™ bitplane. The STM bitplane is used to presentresidual intensities as will be discussed afterwards. The largest timeslot of the one unit time can be 1/(2−log(1−ε)) to be able to correctlydisplay the specific image component. In other words, to use the fewestnumber of bitplanes to correctly display the desired image, eachbitplane time should be 1/(2−log(1−ε)) of the time remaining. Forexample, after setting the MSB bitplane time to 1/(2−log(1−ε)), thesecond bitplane would be set to 1/(2−log(1−ε)) of the remaining time.For example, the time slot for each bitplane is desired to be ⅓ of thetotal left time when c is 0.6321. Specifically, the time slots for thecalibrated bitplanes can be, from the last (LSB) to the first (MSB), 2,1, 3/2, 9/4, 27/8, 81/16 . . . . In other examples, the largest timeslot of the one unit time can be calculated in many other ways. Forexample, the largest time slot can be (1−ε)/(2−ε). With the calculatedframe gain and display time, bitplanes can be calculated from each colorimage component so as to include the intensity variation due to theoff-state light recycling (step 154). This bitplane calculation involvesmany steps; and an exemplary calculation process is illustrated in FIG.7.

Referring to FIG. 7, pixel data of the incoming image frame iscalibrated at step 166 by multiplying the pixel data by the determinedframe gain g_(frame) due to the off-state recycling. The underlyingtheory of such calibration can be obvious from the above descriptionwith reference to equation 1. It is noted that the calibrated pixel datarepresent a calibrated intensity that includes the intensity due torecycled off-state light that may vary over different bitplanes of theimage frame. Such calibrated intensity is the intensity to beaccomplished through a set of bitplanes, which will be detailedafterwards. As an example, FIG. 8 plots an exemplary calibrated pixeldata curve for a particular color image component of an image frame.

Referring to FIG. 8, it is assumed that the image frame has 8-bitsdepth. The image frame, as well as each color image component thereof,therefore has 256 different gray levels. Accordingly, pixel data of eachcolor image component has up to 256 different values ranging from 0 to255. In other examples, the image frame may have other image depths,such as 4 bits, 6 bits, 10 bits, and 12 bits. For demonstration purposeonly, FIG. 8 further assumes that the frame gain g_(frame) is 1.8. Thevertical axis (i.e. Y axis) plots the calibrated pixel data of theparticular pixel color image component (e.g. red color image component)of the image frame, whereas the calibrated pixel data is scaled withinthe range from 0 to the frame gain g_(frame)[0, 1.8]. The horizontalaxis (i.e. X axis) plots indices of image pixels having the pixel dataplotted in the vertical axis with the image pixels being sorted based ontheir calibrated pixel data values. Specifically, image pixels 0 (theupper-left corner image pixel) through P (the bottom-right corner imagepixel) are sorted into image pixel groups based on their calibratedpixel data. Each sorted image pixel group (which comprises one or moreimage pixels) is assigned to an index ranging from 0 to 1. As such, eachindex (and the X axis value) is associated with the pixel(s) of an imagepixel group having the same calibrated pixel data (Y axis value). In analternative example, X axis can plot a cumulative histogram of all imagepixels, sorted by their calibrated pixel data. For example, an X-valueof 0.35 means that 35% of the image pixels in the color image componentare below the corresponding Y value (pixel data value). The principlesof calculating the bitplanes as discussed below is also applicable tothe instance wherein the X-axis plots the cumulative histogram, and toother equivalent variations. Given the calibrated pixel data and thecalibrated pixel data curve, the set of bitplanes are calculated. Thebitplanes can be calculated sequentially from the most-significant-bit(MSB, the first bitplane) to the least-significant-bit (LSB, the lastbitplane), which will be detailed in the following with reference to aparticular example wherein the desired image has 8-bits of grayscale. Itwill be appreciated by those skilled in the art that the followingdiscussion is for demonstration purpose, and should not be interpretedas a limitation. Instead, images with any grayscale depths are alsoapplicable. It is further noted that the bitplanes can be in any desiredformat, such as binary and non-binary.

Referring again to FIG. 7, calculation of the bitplanes following thecalibration of the pixel data (step 166) can start from a calculation ofthe MSB (step 168), which is better demonstrated in FIG. 9. Referring toFIG. 9, curve 179 plots the calibrated pixel data as discussed abovewith reference to FIG. 8. Gain curve 180 plots the weight of the MSBbitplane to be calculated for the specific color image component, whichcan be expressed as equation 6:

$\begin{matrix}{{weight} = {{time}_{bitplane}\frac{1}{1 - {ɛ\left( {1 - x_{i}} \right)}}}} & \left( {{Eq}.\mspace{14mu} 6} \right)\end{matrix}$In equation 6, x_(i) is the ratio of the number of ON-state pixels tothe total number of pixels in the i^(th) bitplane; time_(bitplane) isthe time allotted to the i^(th) bitplane as discussed above withreference to step 153 in FIG. 5; and ε is the off-state light recyclingefficiency. When all pixels are at the ON-state or when the recyclingefficiency is zero, there is no off-state recycling. The weight of thebitplanes as expressed in equation 6 represents the effective brightness(or resulted illumination intensity) of the i^(th) bitplane. It can beobserved from equation 6 that the weight of the bitplane may vary overimage frames even for the same indexed bitplanes (e.g. the MSB and LSBbitplanes) in different image frames.

Based on the calibrated pixel data curve and gain curve of MSB bitplane,the MSB bitplane is calculated according to the pixel data and an upperthreshold. Specifically, the MSB bits of the image pixels (plotted inthe X axis as shown in FIG. 8) having pixel data equal to or higher thanthe upper threshold are set to the ON-state (e.g. “1”), while the MSBbits of the image pixels having pixel data lower than the threshold areset to the OFF-state. In an example, the upper threshold is the point atwhich the calibrated pixel data is equal to or higher than thebit-weight given by the gain curve (180). An example of such upperthreshold is the crossing point (182) of calibrated pixel data curve 179and gain curve 180. All pixels (i.e. pixels in region 184) having thecalibrated pixel data equal to or above the upper-threshold are set tothe ON-state. In the instance when the gain curve (weight curve) and thecalibrated pixel data curve have multiple intersections, theupper-threshold can be selected from the multiple intersections suchthat the most pixels are turned on with such upper-threshold. It isnoted because each coordinate in the horizontal axis is associated withone or a group of image pixels as discussed above, turning on the pixelsin region 184 can be accomplished through associations of the horizontalcoordinates to the individual image pixels. After determining the MSB(the first) bitplane, other bitplanes are sequentially calculated.

Referring again to the flow chart in FIG. 7, the calibrated pixel datais adjusted (step 170) to eliminate the energy (intensity) carried bythe above determined MSB bitplane before calibrating other bitplanes.Specifically, such adjustment is performed by subtracting the energycarried by the MSB bitplane from the calibrated pixel data, asdemonstrated in FIG. 10. As shown in FIG. 10, curve 186 plots thecalibrated bitplane data curve (179 in FIG. 9) after the adjustment.This adjusted calibrated-pixel-data curve can then be used to calibratethe next (the second) bitplane (step 172 in FIG. 7), which is betterillustrated in FIG. 11.

Referring to FIG. 11, curves 186 and 188 are the re-sortedcalibrated-bitplane data curve after the adjustment (as shown in FIG.10) and the gain curve (weight curve) of the second bitplane to becalculated. The intersection of the two curves defines anupper-threshold, as well as region 192, for the second bitplane. Imagepixels in region 192 are set to the ON-state for the second bitplanesuch that all image pixels in region 192 have calibrated pixel data(after the adjustment) equal to or larger than the upper-threshold.After calculation of the second bitplane, the pixel data curve isre-adjusted to eliminate the energy represented by the calculated secondbitplane (step 174 in FIG. 7). The adjusted pixel data curve is shown ascurve 194 in FIG. 12. Calculation of the next (the third) bitplane canthen be conducted in a similar way for the first and second bitplanes asdiscussed above. As shown in the flow chart in FIG. 7, calculation ofthe bitplane (step 172) and adjusting the calibrated pixel data based onthe previously calculated bitplane (step 174) are repeated until thelast bitplane (e.g. the LSB bitplane) is calculated (step 176). Fordemonstration purpose, FIG. 13 through 18 demonstrate calculation of thethird bitplane through the 8^(th) bitplane following the calculationsfor the first and second bitplanes as discussed above.

Referring to FIG. 13, calibrated pixel data curve 194 after adjustmentand re-sorting intersects with gain curve 196; and the intersectiondefines an upper-threshold and region 200 for the third bitplane. Allimage pixels in region 200 are set to the ON-state such that all pixelsin region 200 have calibrated pixel data after the adjustment that areequal to or larger than the upper-threshold. The calibrated pixel datacurve is then adjusted to eliminate the energy represented by the abovedetermined third bitplane. The adjusted pixel data after re-sorting isshown in curve 204 in FIG. 14. Calculation of the next (fourth) bitplanecan then be performed.

Referring to FIG. 14, pixel data curve 204 and weight curve 202 of thefourth bitplane have an intersection; and the calibrated pixel datavalue of such intersection can be defined as a threshold for the fourthbitplane. All image pixels of the fourth bitplane in region 208 are setto the ON-state such that all pixels in region 208 have calibrated pixeldata after the adjustment that are equal to or larger than theupper-threshold. The calibrated pixel data curve is then adjusted toeliminate the energy represented by the above determined fourthbitplane. The adjusted pixel data after re-sorting is shown in curve 210in FIG. 15. Calculation of the next (fifth) bitplane can then beperformed, as demonstrated in FIG. 15.

Referring to FIG. 15, pixel data curve 210 and weight curve 212 of thefifth bitplane have an intersection; and the calibrated pixel data valueof such intersection can be defined as a threshold for the fifthbitplane. All image pixels of the fifth bitplane in region 216 havingcalibrated pixel data after the adjustment equal to or larger than theupper-threshold are set to the ON-state. The calibrated pixel data curveis then adjusted to eliminate the energy represented by the abovedetermined fifth bitplane. The adjusted pixel data after re-sorting isshown in curve 218 in FIG. 16. Calculation of the next (sixth) bitplanecan then be performed, as demonstrated in FIG. 16.

Referring to FIG. 16, pixel data curve 218 and weight curve 220 of thesixth bitplane have an intersection; and the calibrated pixel data valueof such intersection can be defined as a threshold for the sixthbitplane. All image pixels of the sixth bitplane in region 224 havingcalibrated pixel data after the adjustment equal to or larger than theupper-threshold are set to the ON-state. The calibrated pixel data curveis then adjusted to eliminate the energy represented by the abovedetermined sixth bitplane. The adjusted pixel data after re-sorting isshown in curve 226 in FIG. 17. Calculation of the next (seventh)bitplane can then be performed, as demonstrated in FIG. 17.

Referring to FIG. 17, pixel data curve 226 and weight curve 228 of theseventh bitplane have an intersection; and the calibrated pixel datavalue of such intersection can be defined as a threshold for the seventhbitplane. All image pixels of the seventh bitplane in region 232 havingcalibrated pixel data after the adjustment equal to or larger than theupper-threshold are set to the ON-state. The seventh bitplane can thenbe obtained by converting the adjusted pixel data. The calibrated pixeldata curve is then adjusted to eliminate the energy represented by theabove determined seventh bitplane. The adjusted pixel data afterre-sorting is shown in curve 234 in FIG. 18. Calculation of the next(eighth) bitplane can then be performed, as demonstrated in FIG. 18.

Referring to FIG. 18, pixel data curve 234 and weight curve 236 of theeighth bitplane have an intersection; and the calibrated pixel datavalue of such intersection can be defined as a threshold for the eighthbitplane. All image pixels of the eighth bitplane in region 240 havingcalibrated pixel data after the adjustment equal to or larger than theupper-threshold are set to the ON-state. The eighth bitplane can then beobtained by converting the adjusted pixel data.

After calibrating all bitplanes (e.g. the above eight bitplanes as anexample), the calibrated pixel data may have a residual value (energy)that are not represented by the combination of the previously calibratedbitplanes, as shown in curve 244 FIG. 19. This residual energy can becalculated in step 156 in the flow chart of FIG. 5 by subtracting theenergy carried by the eighth bitplane from the calibrated pixel datacurve (134) shown in FIG. 18.

To present this residual energy (step 158 in the flow chart in FIG. 5),a spatial-temporal-multiplexing (STM) technique, such as that set forthin U.S. Pat. No. 7,075,506 to Morgan, issued Jul. 11, 2006, the subjectmatter being incorporated herein by reference in its entirety, can beemployed. Specifically the STM technique generates a bitplane byprobabilistically setting each pixel to either the ON or OFF statedepending upon the pixel data values, the STM mask, and the STM bitweight that is further determined based on the number of pixels beingturned on. An example of such generated STM bitplane is demonstrated incurve 242 as shown in FIG. 20.

Referring to FIG. 20, the weight of the STM bitplane (242) is chosensuch that the total light intensity of the STM bitplane, which isrepresented by the area of block 246, is substantially equal to the areaencompassed by the residual biplane data curve 244 and the horizontalaxis with such area representing the energy of the residual calibratedpixel data. The height of block 246 is then used as the STM bit weight,and pixels of the STM bitplane are probabilistically set to either ON orOFF state depending on the pixel data values, the STM mask, and thiscalculated STM bit weight.

Till this point, bitplanes including the calibrated bitplanes and theSTM bitplanes have been obtained. These bitplanes are capable ofrepresenting both the intensity of the bitplanes derived from thedesired image and the intensity of the recycled off-state light.Presenting these bitplanes in the pixels of the spatial light modulatorof the display system can eliminate the intensity variation problem asdiscussed above.

The above discussed bitplane derivation is only one of many possibleways. In another example, bitplane thresholds values (or equivalentlythe input pixel code levels corresponding to the bitplane thresholdvalues) for all bitplanes (e.g. from the MSB bitplane to the LSBbitplane) can be calculated using the above discussed method. Thecalculated thresholds are stored, for example in a column of a tablewith each threshold being in a cell of the column. Pixel data of allpixels of the image are then compared to the stored thresholds; andvalues of individual pixels in each bitplane are determined based on thecomparison. Taking the MSB bitplane as an example, pixel data of theimage are compared to the calculated threshold for the MSB bitplane.Pixels of the image having the pixel data values equal to or higher thanthe threshold for the MSB bitplane are identified through thecomparison. The MSB bits of such identified pixels of the image are setto the ON-state value, such as “1;” and the MSB bits of other pixels(having image values lower than the threshold for the MSB) are set tothe OFF-state value, such as “0.” Bit values of other pixel data can bedetermined in the same way using their corresponding thresholds. Theadjusted pixel data can then be converted to corresponding bitplanesusing many ways, such as those set forth in U.S. Pat. No. 7,075,506 toMorgan, issued Jul. 11, 2006, U.S. Pat. No. 5,663,749 to Farris issuedSep. 2, 1997, and U.S. patent application Ser. No. 10/648,608 toRichards filed Aug. 25, 2003, the subject matter of each beingincorporated herein by reference in its entirety. It will be appreciatedby those of skill in the art that the point at which image pixels areconverted to bitplane pixels (whether immediately following eachbitplane threshold calculation or after all bitplane thresholdcalculations are complete) has no effect on the function or spirit ofthe present invention. In an example, the upper threshold is the pointat which the calibrated pixel data is equal to or higher than thebit-weight given by the gain curve (180). An example of such upperthreshold is the crossing point (182) of calibrated pixel data curve 179and gain curve 180. All pixels (i.e. pixels in region 184) having thecalibrated pixel data equal to or above the upper-threshold are set tothe ON-state.

In another example, a look up table (LUT) can be created and used forderiving individual bitplanes with an exemplary LUT being shown inTABLE 1. For demonstration purposes, table 1 assumes that the inputimage has 256 pixel levels and n bitplanes are to be derived for theimage. Different pixel data levels for each color image component are inrows of the table 1; while the bits from MSB to LSB of the bitplanes arein columns of the table. The value in each cell indicates the ON (“1”)or OFF-state (“0”). This table can be generated through following steps.

After determining the thresholds for each bitplane (of each color imagecomponent) as discussed above with reference to FIG. 8 through FIG. 20,the most possible values of pixel data levels (from 0 to 255 when theinput image has 256 levels) for each bitplane can be determined based onthe pixel data value of the threshold (Y coordinate of the threshold).As an example wherein the threshold for the MSB is determined as shownin FIG. 9, the threshold is determined at cross-point 182. The Ycoordinate (gained pixel data values) splits the pixel data (for theMSB) bitplane into two sections—the first section having pixel dataequal to or higher than the Y coordinate; and the second section havingpixel data lower than the Y coordinate. The MSB bits corresponding toall pixels in the first section are set to the ON-state value (“1”); andthe MSB bits corresponding to all pixels in the second section are setto the OFF-state value (“0”). For example, assuming the Y coordinate ofpoint 182 in FIG. 9 is 0.84, which corresponds to 117.6 in the [0, 255]image data levels, then the most probable level L_(p) can be set to 117or 118, or other approximate integer levels. Assuming the most probablelevel is 117, all bits in the MSB for the pixels having pixel datalevels below 117 (i.e. levels from 0 to 116) are set to the OFF-statevalue (“0”); while all bits in the MSB for the pixels having pixel datalevels equal to or higher than 117 (i.e. levels from 117 to 255) are setto the ON-state value (“1”), resulting in the values in the MSB column.Other bit planes values can be calculated in the same way with theirindividual thresholds being calculated using the method as discussedabove with reference to FIG. 8 to FIG. 20. Therefore, it can be seenthat table 1 includes the effect of the recycled off-state light; andcan thus be different from frame to frame due to different off-statelight being recycled. The bitplanes can be determined using the LUTshown in table 1, as discussed in the following.

For a given pixel with a value from 0 to n, its corresponding bitplanebit values can be immediately determined from the LUT, as shown inTABLE 1. For example, assuming the red channel pixel value of the i^(th)pixel in the incoming image is 3, then the red MSB bit for the i^(th)pixel is 0; and the bits of the third red bitplane and the red LSB ofthe i^(th) pixel are 1. In another example, assuming the red channelpixel value of the p^(th) pixel in the incoming image is 255, then bitsof all bitplanes from the red MSB to LSB are 1 according to theexemplary table 1. As such, bitplanes can be determined for all pixelvalues of the incoming image based on the LUT. It is noted that theabove discussion, especially table 1, is for demonstration purpose only;and table 1 is only an example. The values of bitplanes in table 1 canbe different for different images or image frames of video(s). Thelayout of the LUT, for example, the numbers of columns and rows andtheir arrangements in rows and columns, can be changed. For example, thedata values can be plot in the columns; while the bits of the samebitplane can be plotted in rows. Input data values may or may not beplotted sequentially in ascending or descending orders; and canalternatively be in any orders, even randomly, so as the bits of thebitplanes in the table.

TABLE 1 Bitplane ON/OFF Input Data Residual Values MSB 1 2 3 . . . . . .LSB n (float) Red 0 0 (off) 0 0 0 0 0 1 0 0 0 0 1 0 2 0 0 1 0 1 0 3 0 01 0 1 1 . . . 0 1 1 0 1 1 . . . . . . . . . . . . . . . . . . . . . 255 1 (on) 1 1 1 1 1 Green 0 0 0 0 0 1 0 0 1 0 2 1 0 1 0 3 1 0 1 1 . . . 1 01 1 . . . . . . . . . . . . . . . . . . . . . 255  1 0 1 1 Blue 0 1 2 3. . . . . . . . . . . . . . . . . . . . . . . . 255 For demonstration purpose, a pseudo-code for deriving bitplanes based onthe thresholds and a LUT is presented in appendix A.

In the above, derivation of the set of bitplanes from a specific colorimage component of the desired image and determination of a S™ bitplanehave been accomplished. These processes can be repeated for all otherimage color components of the desired image.

As an example, FIG. 21 a and FIG. 21 b show two pictures to demonstratethe effect of the brightness boost. Specifically, the picture in FIG. 21a is produced by a display system with substantially zero off-staterecycling efficiency. The picture in FIG. 21 b is produced by the samedisplay system with non-zero off-state recycling efficiency and thebitplanes derived as discussed above.

Another Pulse-Width-Modulation Algorithm

As discussed above, because different bitplanes (either having or nothaving the same significance) can cause different numbers of on-stateand off-state pixels of the spatial light modulator, the amount ofrecycled off-state light varies over bitplanes for a given off-statelight recycling mechanism. Such variation in turn causes unwantedillumination intensity changes on the screen in reproducing the desiredimage. As a consequence, the bitplanes derived from the desired imagewhen displayed by the pixels of the spatial light modulator do notresult in the desired image on the screen. This problem canalternatively be solved by displaying the bitplanes at different clockspeeds depending upon the number of off-states (or on-states) in thebitplanes. Specifically, bitplanes having different numbers of off-statepixels therein are displayed at different clock speeds in the spatiallight modulator such that bitplanes having different number of off-statepixels (resulting in different amounts of recycled off-state light) canbe displayed for different time periods so as to compensate for theillumination intensity variations due to the off-state light recycling.This bitplane displaying algorithm can result in that bitplanes ofdifferent image frames in a video content composed of image frames maybe displayed for different time periods, even for the bitplanes with thesame significance.

As a way of example, FIG. 22 a through FIG. 22 c schematicallyillustrate a method for displaying bitplanes at different clock speedsdepending upon the number of off-states (or on-states) in the bitplanes.

Referring to FIG. 22 a, an exemplary sequence of color duty cycleswithin a frame time period is schematically illustrated therein. In thisexample, the sequence of color duty cycles comprises red, green, andblue colors with each color being allotted a display time such that thetotal display time of all colors is substantially equal to the frametime period. For example, the red, green, and blue colors canrespectively be 33%, 32%, and 35% of the entire frame time. Each colorduty cycle time is scaled down by a scaling factor that is determinedbased on the gain due to off-state light recycling in displaying thecorresponding color channel, as illustrated in FIG. 22 b. Specifically,the red, green, and blue color duty cycle time periods are respectivelyscaled down by scaling factors S_(r), S_(g), and S_(b). The scalingfactors S_(r), S_(g), and S_(b) are determined based on the gainsg_(red), g_(green), and g_(blue) that are calculated using equation 4 asdiscussed above. In one example, the scaling factors can be set to thecorresponding gains, that is, S_(r)=g_(ref); S_(g)=g_(green); andS_(b)=g_(blue). In another example, the scaling factors can beproportional to the gains, which can be expressed as: S_(r)=a×g_(red);S_(g)=a×g_(gree); and S_(b)=a×g_(blue), wherein a can be a value that isless or greater than 1 but the resulted scaling factors are larger than1.

After scaling, the sequence of color duty cycles is compressed,resulting in a residual time period in the frame time period as shown inFIG. 22 b. The compressed sequence of color duty cycles in FIG. 22 b isthen uniformly stretched out by a stretching factor E so as to fit theentire frame time period as illustrated in FIG. 22 c. The stretchingfactor E can be calculated using the following equation 7.E=ε(Σt _(i) A _(i))+(1−ε)  (Eq. 7)In equation 7, t_(i) is the proportion of the time allotted to eachcolor image component of the desired image (or image frame).Specifically, t, is defined as T_(i)/T_(frame), wherein T_(i) is thetime allotted to the i^(th) color image component (e.g. red color imagecomponent); and T_(frame) is the frame time. Accordingly, Σt_(i)=1.A_(i) is the average pixel value of the i^(th) color image component inlinear light space after de-gamma has been applied. Σ(t_(i)A_(i))represents the average total number of on-state pixels in the imageduring the frame period T_(frame).

In display applications, compression of the color duty sequence asillustrated in FIG. 22 b can be achieved by designing thepulse-width-modulation sequence. A PWM sequence is referred to as aprocess of displaying the bitplanes derived from the image based on thePWM by the pixels of the spatial light modulator. In an example whereindisplaying the bitplanes is accomplished through multiple steps, such asloading the bitplanes into the pixels of the spatial light modulator andresetting the pixels to states corresponding to the loaded bitplanes,the PWM sequence is referred to the process having the steps of loadingthe bitplanes and resetting the pixels of the spatial light modulators.Stretching the color duty cycles as illustrated in FIG. 22 c can beaccomplished through adjusting the clock speed of the spatial lightmodulator of the display system, which will be detailed in thefollowing. It is noted that FIG. 22 a through FIG. 22 c are fordemonstration purpose wherein the color duty cycle comprises red, green,and blue colors. In general, the color duty cycle may comprise anycombinations of colors that are preferably, tough not required, selectedfrom red, green, blue, yellow, cyan, magenta, and white.

As an example, FIG. 23 is a diagram schematically illustrating anexemplary operation for displaying bitplanes at different clock speeds.Referring to FIG. 23, pixel data of a desired image is received by dataformatter 250, which can be a spatial-temporal-multiplexing unit (STM),for formatting the received pixel data into bitplanes 252. An exemplarydata formatter is set forth in U.S. Pat. No. 7,075,506 to Morgan, issuedJul. 11, 2006, the subject matter being incorporated herein by referencein its entirety. Other data formatting algorithms and technologies thatconvert image pixel data to bitplane data (or bitplanes) are alsoapplicable. The pixel data can be in any desired formats, such as RGBdata, CYM data, and YPbPr data. The formatted bitplanes can also be inany desired formats, such binary formats and non-binary-formats. Whenthe input image is a color image having different color imagecomponents, a set of bitplanes is derived from each color imagecomponent. The derived bitplanes of the image are delivered to pixels ofspatial light modulator 108; and the states (e.g. the ON and OFF-statefor binary pixels) of the pixels are then set based on the values of theindividual pixels in the delivered bitplanes.

In general, a display system has at least one clock that generates clocksignals based on which operations of the functional members includingpixels of the spatial light modulator of the display system can besynchronized; or operations of a functional member of the display systemcan be triggered. Accordingly, durations of the bitplanes in the pixelsof the spatial light modulator can be varied by changing the clockspeed. As a way of example wherein a bitplane is to be displayed for 4time units at the first clock speed according its weight (significance),such bitplane however can be actually displayed for 8 time units at thesecond clock speed that is half the first clock speed; and can bedisplayed for 2 time units at the third clock speed that is twice thefirst clock speed.

For displaying bitplanes at different clock speeds while within a frameperiod, a pulse-width-modulation (PWM) sequence is designed such thatthe sequence can run in a time period T of T_(frame)×I_(o)/I_(max),which is shorter than the frame time T_(frame), such as 1/(60 HZ);wherein I_(o) is the intensity defined in equation 1; and I_(max) is themaximum value of I as described in equation 1. In an example wherein therecycling efficiency ε is ⅔, I_(max) is 3I_(o); and the time period Tfor the PWM sequence can be 16667/3=5556 microseconds.

With the PWM sequence time that is shorter than the frame time,bitplanes can be displayed at different clock speeds without exceedingthe frame time, wherein the clock speeds can be dynamically adjustedbased on the number of off-state pixels in the bitplanes. As a way ofexample, the display time for each bitplane can be determined byequation 8 as expressed in the following:

$\begin{matrix}{C = \frac{C_{o}E}{1 - {ɛ\frac{n_{off}}{N_{total}}}}} & \left( {{Eq}.\mspace{14mu} 8} \right)\end{matrix}$In equation 8, C is the adjusted clock speed; C_(o) is the clock speedprovided by the display system to the spatial light modulator; n_(off)is number of off-state pixels in the bitplane or bitplanes (whenmultiple bitplanes are displayed at a time) being displayed by thepixels of the spatial light modulator; N_(total) is the total number ofpixels in the bitplane or bitplanes being displayed; ε is the recyclingefficiency; and E is the time-stretch factor, which can be calculated bythe above equation 7.

It can be observed from equation 8 that clock speed C is larger forbitplanes with more off-state pixels than for bitplanes with fewernumber of off-state pixels. As a consequence, a first bitplane in oneimage frame may be displayed for a shorter time period than a secondbitplane in another image frame when the first bitplane has moreoff-state pixels than the second bitplane even though the first andsecond bitplanes have the same weight (significance).

Calculating the clock speed and operating the spatial light modulator atthe calculated clock speed can be accomplished in many ways, one ofwhich is schematically illustrated in FIG. 23. As shown in FIG. 23,average pixel value calculator 254 is input pixel data of incomingimages for calculating the average pixel value A_(i) for each colorimage component of the input image; and forwarding the calculated A_(i)to clock speed calculator 258. Before calculating the adjusted clockspeed for the spatial light modulator, the number of off-state pixels ineach bitplane is calculated by real-time bit-counter 256. As a way ofexample wherein the PWM sequence comprises a step of loading thebitplanes into pixels of the spatial light modulator and a step ofresetting the pixels to states corresponding to the loaded bitplanes,the pixels of the spatial light modulator are divided into a multiplereset groups. A set of groups of bitplanes is derived from the desiredimage with each bitplane corresponding to a reset group. The bitplanesare loaded to the pixels of the spatial light modulator in reset groups;and the pixels are reset according to the loaded bitplanes. Fordemonstration purpose, FIG. 24 schematically illustrates an array ofpixels of a spatial light modulator with the pixels in the array beingdivided into reset groups.

Referring to FIG. 24, pixel array 270 of a spatial light modulator, suchas spatial light modulator 108 in FIG. 1, comprises an array ofindividually addressable pixels (e.g. pixel 274) that can bemicromirrors, liquid-crystal-on-silicon, and other types of devices. Thepixels are connected to a set of reset lines, such as reset lines MBRST0 266 through MBRST P 268, through each of which reset signals can bedelivered to the pixels. Upon receipt of the reset signals, the pixelsare set to desired states. In this example, the entire pixel array isdivided into multiple reset groups (e.g. reset groups 272 and 276); andeach group is connected to a reset line. For example, pixels of resetgroup 272 are connected to reset line MBRST 0 266; and pixels of resetgroup 276 are connected to reset line MBRST P 268.

The pixels of the pixel array can be divided in many ways. For example,the pixels can be divided into groups with equal number of rows (orcolumns, or pixels or sub-blocks of pixels). As shown in the figure,assuming the pixels array has M rows, the pixel array can be equallydivided into M/(i+1) groups with each group comprising (i+1) rows and ibeing 31 or any desired values. Pixels of each reset group are connectedto one of a set of reset lines MBRST 0 to MBRST P. The reset lines areconnected to reset driver 164, which can be a DAD reset driver by TexasInstrument, Inc.

Given the pixel reset groups, bitplanes are loaded to the correspondingreset groups of pixels for being displayed. The number of off-statepixels in each bitplane being displayed by the pixels of the spatiallight modulator can then be calculated, as schematically illustrated inFIG. 25. Referring to FIG. 25, at a particular time, a set of bitplanes278 comprising bitplanes 0, 1, 2 through P are loaded and displayed bythe pixels of the spatial light modulator. Each bitplane of the set ofbitplanes 278 corresponds to a reset group of pixels numbered as resetgroup 0, 1, 2, through P, as shown in the figure. It is noted that thebitplanes numbered as 1 through P in this example may or may not havethe same significance. Before counting the number of off-state pixels inthe bitplanes being displayed by the pixels of the spatial lightmodulator, cells of column 1 of the counter table (280) are set toinitial values, such as 0 or any other predetermined values. At eachtime a bitplane is loaded to a group of pixels of the spatial lightmodulator, the value of the corresponding cell in column 1 isincremented by 1 for each off-state pixel in the loaded bitplane. Takingan example wherein the reset group 0 of bitplane 0 has 20 off-statepixels, the cell corresponding to reset group 0 in column 1 of thecounter table (180) has a value of 20 after bitplane 278 is loaded tothe reset group 0 of the spatial light modulator. When the pixels of thereset groups (e.g. reset group 0) are reset based on the loadedbitplanes (e.g. bitplane 0), all values in the cells of column 1 ofcounter table 180 are shifted to the corresponding cells in column 0followed by resetting the values of the cells in column 1 to theirinitial values. By summing up the values of cells in column 0, the totalnumber of off-state pixels of the bitplanes being displayed by thepixels of the spatial light modulator can be obtained. This value isdelivered to clock speed calculator 258 in FIG. 23.

In another example wherein a set of bitplanes is derived from thedesired image with each bitplane being a collection of all pixel databits of the same significance, calculation of the number of off-statepixels of a bitplane being displayed can also be performed using thecounter table (180). Specifically, the counter table may be adapted suchthat each of columns 1 and 0 has one cell. After loading a bitplane tothe entire array of pixels of the spatial light modulator, the cell incolumn 1 of the counter table (280) is incremented by 1 from its initialvalue for each off-state in the loaded bitplane. After the pixels of thespatial light modulator are reset based on the loaded bitplane, thevalue of the cell in column 1 can be shifted to the cell in column 0followed by resetting the cell in column 1 to its initial value. Theresulted value in the cell in column 0 is the total number of off-statepixels in the bitplane being displayed. Such obtained total number ofoff-state pixels is delivered to the clock speed calculator in FIG. 23.

It is noted that the above described off-state pixel calculation methodusing a counter table is only one of many possible ways. Many othermethods using different counter tables or even without using a countertable are also applicable.

Referring again to FIG. 23, clock speed calculator 258 calculates theadjusted clock speed for the spatial light modulator using equations 7and 8 with the calculated average pixel values A, and the number ofoff-state pixels n_(off) in the bitplane or bitplanes being displayed.The calculated speed is sent to clock adjustor 260 that is capable ofpicking up the clock signals for the spatial light modulator andadjusting the clock signals based on the calculated speed from clockspeed calculator 258. The adjusted clock speed is delivered tocontroller 262 that controls operations of the spatial light modulator(108) based on the received clock speed from clock adjustor 260.

As an example, FIG. 26 illustrates a flow chart having steps executedfor displaying bitplanes at different clock speeds. Referring to FIG.26, a set of pixel data of an image or an image frame of a sequence ofvideo frames is received at step 282. The pixel data are formatted to aset of bitplanes according to a pre-determined bitplane format, such asbinary and non-binary bitplane formats (step 284). Each set of bitplanescorresponds to a color image component when the input image is a colorimage. The set of bitplanes is then displayed by the pixels of thespatial light modulator at different clock speeds (step 286), whichfurther comprises steps 288 through 294. At step 288, the average pixelvalue A_(i) of each color image component (when the input image is acolor image) or the average pixel value of the input grayscale image;the total number of off-state pixels n_(off) in the bitplane orbitplanes being displayed by the pixels of the spatial light modulator;and the time stretch factor E are calculated. Given the above calculatedaverage pixel value, total number of off-state pixels, and thetime-stretch factor E, the clock speed for displaying the specificbitplane or a specific group of bitplanes (e.g. group 278 in FIG. 25) bythe spatial light modulator is calculated based on equations 7 and 8.The clock speed to which operations of the spatial light modulator aresynchronized to is adjusted at step 292. The adjusted clock speed isthen applied to the spatial light modulator for displaying the specificbitplane. The above displaying process (step 286) is repeated for allbitplanes for each color image component; and for all bitplanes of allcolor image components. The process flows back to step 282 for the nextinput image.

It is noted that calculation of the average pixel value is independentfrom other calculations until the calculation of the time-stretch factorE at step 288, and can alternatively be performed at any time prior tostep 288 wherein the time-stretch factor E is calculated but after step282 wherein pixel data are received. For example, the average pixelvalue can be performed before formatting the pixel data into bitplanes(step 284) and after receipt of the pixel data (step 282).

The method as discussed above with reference to the flow chart in FIG.26 is not only applicable to color duty cycles that are composed ofprimary colors (there is no color that is a combination of other colorsin the sequence), but is also applicable to any other color duty cyclesthat are composed of secondary colors (a color that is a combination ofother colors in the sequence), such as a color duty sequence of red (R),green (G), blue (B), cyan (C), yellow (Y), magenta (M), and white (W)colors. The secondary colors in a color duty cycle sequence have manyadvantages than color duty cycle sequences otherwise. This arises fromthe fact that the secondary colors in a color duty sequence are oftenobtained by overlapping the illumination time periods of illuminators inthe display system. FIG. 27 schematically illustrates an exemplaryoverlapping scheme by assuming that the display system employsilluminators emitting red, green, and blue colors. It will beappreciated by those skilled in the art that the following discussion isfor demonstration purpose, and should not be interpreted as alimitation. Many other variations within the scope of this disclosureare also applicable. For example, the display system may employilluminators capable of emitting other colors, such as colors selectedfrom red, green, blue, yellow, cyan, magenta, and white. The secondarycolors may also be obtained by many other ways, such as a color filtercomprising a set of transparent color segments that comprise thesecondary colors.

Referring to FIG. 27, during a frame time period, red, green, and bluecolors exhibit overlaps resulting in a sequence of color duty cyclescomprising secondary colors of yellow, cyan, magenta, and white. As aconsequence of the overlap, the total ON-time of the R, G, and Billuminators is greater than the frame time period, which results inbrighter full-on white when the sequence of color duty cycles is appliedto illuminating the spatial light modulator for projecting desiredimages on a screen. The profile of the color duty cycle sequence,including relative positions and time periods of colors in the sequence,is determined by the sequence and duration of turning on and turning offthe individual illuminators. In general, the sequence can be of anydesired profiles. For example, the sequence can have a profile that issubstantially determined by the BrilliantColor™ algorithm from TexasInstruments, Inc. Given the color duty sequence, bitplanes for eachcolor of the sequence are derived and displayed at different clockspeed, as schematically illustrated in FIG. 28 a through FIG. 28 c.

Referring to FIG. 28 a, the color duty cycle of magenta (M), red (R),yellow (Y), white (W), green (G), cyan (C), and blue (B) fits in a frametime period. The time periods of the color duty cycles are respectivelyscaled down based on scaling factors S_(m), S_(r), S_(y), S_(w), S_(g),S_(c), and S_(b) as illustrated in FIG. 28 b. The scaling factors aredetermined based on the gains calculated from the off-state recycling ofthe bitplanes associated with the color duty cycles, as discussed abovewith reference to FIG. 22 a through FIG. 22 c. The scaled sequence ofduty cycles are then uniformly stretched by stretching factor E so as tofit into the entire frame time period, as illustrated in FIG. 28 c.

Similar to that discussed above with reference to FIG. 22 a through FIG.22 c, bitplanes are derived from the desired image for each color of thesequence of color duty cycles illustrated in FIG. 28 a. Compression ofthe color duty sequence as illustrated in FIG. 28 b can be achieved bydesigning the pulse-width-modulation sequence such that thepulse-width-modulation sequence can be operated within a time T that issubstantially T_(frame)×I_(o)/I_(max), wherein I_(o) is the intensitydefined in equation 1; and I_(max) is the maximum value of I asdescribed in equation 1. Each color duty cycle can then be locallystretched to the exact desired time through adjusting the clock speed ofthe spatial light modulator. Globally stretching the color duty cyclesas illustrated in FIG. 28 c can be also accomplished through adjustingthe clock speed of the spatial light modulator of the display system.The derived bitplanes can then be displayed by the spatial lightmodulator based on the stretched color duty cycles using the clockdropping technique so as to obtain brightness enhanced image on thescreen.

As an alternative feature, the brightness can be further improved byre-distributing energies among the colors of the sequence of color dutycycles while maintaining the color consistency of the image on thescreen but with enhanced brightness. This arises from the fact that amixture of primary colors (e.g. R, G, and B) can be represented by thecorresponding secondary color (or white). For example, a mixture of thesame amount of R, G, and B colors can be represented (replaced) by whitecolor. A mixture of R and G colors can be represented (or replaced) byyellow color. Therefore, an energy distribution of an image pixel over acolor duty cycle having secondary colors and white can be equivalentlyrepresented by another energy distribution over the same color dutycycle sequence. For enhancing the brightness of the displayed image,energy distribution can be performed in favor of the energy in the whiteand secondary colors, as will be discussed in the following withreference to FIG. 29 a and FIG. 29 b.

Referring to FIG. 29 a, an exemplary energy distribution over a sequenceof color duty cycles for single image pixel n is schematicallyillustrated therein with the duty cycle sequence comprising red, green,blue, cyan, magenta, yellow, and white. The vertical axis plots thepixel value, normalized by the duty cycle of that color, for the colorslisted on the horizontal axis. In the example shown in FIG. 29 a, imagepixel n is sequentially displayed with 100% red color, 76% green, 56%blue, less than 1% cyan, 5% magenta, 0% yellow, and 5% white. The sameunit amount of red, green, and blue colors, which is equal to or lessthan the minimum pixel values of red, green, and blue, can betransferred to the white segment so as to increase the value of thewhite color; and the transferred amount is reduced from the red, green,and blue colors. This transfer process can be expressed in equations 9.W ₁ =W ₀+Min(R ₀ ,G ₀ ,B ₀)R ₁ =R ₀−Min(R ₀ ,G ₀ ,B ₀)G ₁ =G ₀−Min(R ₀ ,G ₀ ,B ₀)B ₁ =B ₀−Min(R ₀ ,G ₀ ,B ₀)  (Eq. 9)In equation 9, W₁ and W₀ are the pixel values of the white color afterand before the energy transfer. Min(R₀,G₀,B₀) is the minimum value ofthe R, G, and B pixel values before the energy transfer. R₁ and R₀ arethe pixel values of the red color after and before the energy transfer.G₁ and G₀ are the pixel values of the green color after and before theenergy transfer; and B₁ and B₀ are the pixel values of the blue colorafter and before the energy transfer.

After the above energy transfer, another same amount of red and greencolors can be transferred to the yellow color, which can be expressed asequation 10.R ₂ =R ₁−Min(R ₁ ,G ₁)G ₂ =G ₁−Min(R ₁ ,G ₁)Y ₁ =Y ₀+Min(R ₁ ,G ₁)  (Eq. 10)In equation 10, Min(R₁,G₁) is the minimum value of the R₁ and G₁ pixelvalues. R₂ and R₁ are the pixel values of the red color after and beforethe energy transfer to yellow. G₂ and G₁ are the pixel values of thegreen color after and before the energy transfer to yellow; and Y₁ andY₀ are the pixel values of the yellow color after and before the energytransfer.

The energy distribution after the above energy transfers isschematically illustrated in FIG. 29 b. Referring to FIG. 29 b, thepixel value of the red color is 24%; and the pixel value of green andblue colors are zero. Cyan color does not change its pixel value. Thepixel value of the yellow color is increased from 0 to 20%; and thepixel value of the white color is increased from 5% to 61%.

The above energy re-distribution is performed for all image pixels. Ifpossible, two additional energy transfers are performed that are similarto that of equation 10, but for cyan and magenta. Specifically, afterany possible yellow transfer has been performed, the minimum of theresulting green and blue pixel value is transferred to cyan.Subsequently, the minimum of the resulting red and blue pixel value istransferred to magenta. Based on the image after the energyre-distribution, bitplanes are derived for image components with colorsin the color duty cycle sequence. Off-state recycling gains arecalculated from the off-state recycling of the image components withcolors in the color duty sequence. The calculated gains are used todetermine the scaling factors for scaling the sequence of color dutycycles and pulse-width-modulation sequence as discussed above withreference to FIGS. 28 a and 28 b.

In re-distributing energies among the colors of the sequence of colorduty cycles, the energy re-distribution may be subject to constraints tomaintain proper image presentations. For example, the amount of energymoved into a particular color duty cycle is desired to be equal to orless than the maximum pixel value allowed by the particular color dutycycle such that the total energy of the particular color duty cycle(including the moved amount) can be achieved by displaying thecorresponding bitplanes. This arises from that the fact that, when thetime period of a particular color duty cycle, such as the white colorduty cycle, is very small the total amount of energy (including themoved energy from the R, G, B colors) may not be properly achieved bydisplaying the corresponding bitplanes during the short duty cycle timeperiod. As a consequence, the amount of energy to be moved to theparticular color, such as white is desired to be determined by themaximum value allowed by the time period of the specific color dutycycle. In general, it is preferred, though not required, that the energyre-distribution preferably starts from moving energies into the whitecolor duty cycle; and move as much energy from the primary colors (e.g.R, G, and B) into the white color duty cycle; and then performing energyredistribution towards other secondary colors. Alternatively, energyredistribution can be performed from moving energies into one or moresecondary colors, then moving energies into the white color duty cycle.

In addition to the above constraint, other constraints may apply, whichare listed as follows. In the following inequalities, r, g, and brepresent the input pixel's RGB values, normalized to a value in therange from 0 to 1; and R, G, B, C, M, Y, and W represent the duty cycleof each color illuminant such that the sum of R, G, B, C, M, Y, and W is1.

g <= G + C + Y + W; r <= R + M + Y + W; b <= B + M + C + W; g − r <= G +C; r − g <= R + M; r − b <= R + Y; b − r <= B + C; g − b <= G + Y; b − g<= B + M; r − g − b <= R; b − r − g <= B; g − r − b <= G; g + b − r <=G + 2C + B + W; g + r − b <= G + R + 2Y + W; r + b − g <= R + B + 2M +W; r >= 0: g >= 0; and b >= 0.

In performing the energy re-distribution, each of the above constraintscan be checked to ensure that no constraints are violated. If aviolation occurs in moving an amount of energies, the amount of energiesto be moved is desired to be adjusted to remove the violation. It isnoted that the above discussion with reference to FIG. 29 a and FIG. 29b are for demonstration purpose, and should not be interpreted as alimitation. Any variations within the scope of this disclosure are alsoapplicable. Specifically, the energy distribution shown in FIG. 29 a isonly an example for a particular image pixel. In fact, an image pixelmay have any energy distributions different from that shown in FIG. 29a.

It will be appreciated by those of skill in the art that a new anduseful off-state light recycling mechanism and a pulse-width-modulationtechnique for use in display systems employing an off-state recyclingmechanism have been described herein. In view of the many possibleembodiments, however, it should be recognized that the embodimentsdescribed herein with respect to the drawing figures are meant to beillustrative only and should not be taken as limiting the scope of whatis claimed. Those of skill in the art will recognize that theillustrated embodiments can be modified in arrangement and detail.Therefore, the devices and methods as described herein contemplate allsuch embodiments as may come within the scope of the following claimsand equivalents thereof.

APPENDIX A Pseudo-code of an exemplary process for deriving bitplanesSTART ε = recycling efficiency for each frame   for each color    Calculate Average Picture Level for each color in linear space    APL(color) = sum(degamma(pixels(color))) / total number of pixels    Find the recycling gain for this color channel     gain(color) = 1 /(1−ε*(1−APL(color)))   end (for each color)   Find the gain for thisframe (the minimum gain of the three color channels)   gain_(frame) =min(gain)   for each color     Initialize LEVEL-COUNT TABLE by creatinga histogram of the input levels     (0..255) along with the linear lightlevel after the frame gain has been applied: Input Pixel Level GainedLinear Level Count 0 = gain_(frame) # pixels at *degamma(input level 0)code 0 1 = gain_(frame) # pixels at *degamma(input level 1) code 1 . . .. . . . . . 255  = gain_(frame) # pixels at *degamma(input level code255 255)     Initialize the time remaining in the frame    time_remaining = 1     for each bitplane from MSB to LSB      Calculate the duration of this bitplane (out of a total time of 1)      bit_time = time_remaining*(1/(2-log(1− ε)))       time_remaining =time_remaining − bit_time       for each input level transition from 0→1to 254→255  (could be         replaced by a smart search algorithm, suchas a binary search,         to find first level above gain_(bitplane))        Determine fraction of pixels below this level transition        fraction_below = cumulative_sum(pixel_count(0 to level)) /    total number of pixels         Determine recycling gain for thisbitplane if this fraction of     pixels are off         gain_(bitplane)= bit_time*(1./(1−(ε*fraction_below)))         if (Gained Linear Levelfor input level > gain_(bitplane))           All pixels with GainedLinear Level at or above this             upper level threshold will beON for this bitplane           Assign ON (1) for all levels of thiscolor above current             level for this bitplane into ENUMERATION            TABLE: Input Bitplane ON/OFF Data MSB Residual Values 1 2 3. . . . . . LSB n (float) Red 0 0 0 0 0 0 0 (off) 1 0 0 0 0 1 0 2 0 0 10 1 0 3 0 0 1 0 1 1 . . . 0 1 1 0 1 1 . . . . . . . . . . . . . . . . .. . . . 255  1 1 1 1 1 1 (on) Green 0 0 0 0 0 1 0 0 1 0 2 1 0 1 0 3 1 01 1 . . . 1 0 1 1 . . . . . . . . . . . . . . . . . . . . . 255  1 0 1 1Blue 0 1 2 3 . . . . . . . . . . . . . . . . . . . . . . . . 255           Update the LEVEL-COUNT TABLE by subtracting the            amount of light displayed during this bitplane            from the Gained Linear Level for each input            level: Input Pixel Level Gained Linear Level Count 0 =previous − gain_(bitplane) (if # pixels at bitplane ON at this level)code 0 = previous (if bitplane OFF) 1 = previous − gain_(bitplane) (if #pixels at bitplane ON at this level) code 1 = previous (if bitplane OFF). . . . . . . . . 255  = previous − gain_(bitplane) (if # pixels atbitplane ON at this level) code 255 = previous (if bitplane OFF)          Sort the LEVEL-COUNT TABLE in ascending order            based on Gained Linear Level, keeping each row            intact.           BREAK from for each level transition loop        else           Assign OFF (0) for this level of this color forthis             bitplane in ENUMERATION TABLE.         end if (GainedLinear Level for input level > gain_(bitplane))       end (for eachinput level transition)     end (for each bitplane)     Use STMdithering to achieve the residual intensities:     The values remainingin the LEVEL-COUNT TABLE after the LSB level has        been calculatedrepresent the intensities to be represented via STM. Bitplane ON/OFFInput Data MSB Values 1 2 3 . . . . . . LSB n Residual (float) Red 0 0 00 0 0 0 = Gained Linear Level (off) from LEVEL-COUNT TABLE after LSB 1 00 0 0 1 0 . . . 2 0 0 1 0 1 0 . . . . . . . . . . . . . . . . . . . . .. . . . . .     Calculate the light intensity of the STM bitplane bydetermining, on average,       how many pixels will be ON. This is afunction of the mean of the       residual values.     APL_(residual) =sum(Gained Linear Levels * pixel count) / total number of pixels    fraction_OFF_(STM) = (APL_(residual) − time_remaining) /(APL_(residual) * ε −       time_remaining)     Intensity_(STM) =time_remaining * (1 / (1− ε*fraction_OFF_(STM)))     Scale the residualvalues by this STM light intensity to get a number between 0       and 1that STM can use for thresholding.     for each Input Data Level      Residual = Residual / Intensity_(STM)     end (for each Input DataLevel)   end (for each color)   With the ENUMERATION TABLE completelyfilled in (including the scaled residual      values), calculatebitplanes and perform STM as normal. end (for each frame)

1. A method of allocating display times comprising: representing imagedata for an image to be displayed during a frame time as a sequence ofone or more bit planes for one or more colors; calculating a recyclinggain for each bit plane; compressing each bit plane based on therecycling gain; determining an unused portion of the frame time based onthe compressed bit planes; and stretching the bit planes to utilize theunused portion of the frame time.
 2. The method of claim 1, comprising:displaying the stretched bit planes.
 3. The method of claim 1, whereinthe one or more colors comprise primary colors.
 4. The method of claim1, wherein the one or more colors comprise primary colors andrepresenting image data comprises receiving image data.
 5. The method ofclaim 1, comprising: receiving the image data representing an input setof colors; and converting the image data to image data representing theone or more colors.
 6. The method of claim 5, wherein converting theimage data comprises converting the image data to image datarepresenting both colors generated by one or more color sourcesindividually and colors generated by one or more color sources incombination.
 7. The method of claim 5, wherein converting the image datacomprises converting the image data to image data representing bothcolors generated by one or more color sources individually and colorsgenerated by one or more color sources in combination, and wherein theimage data representing colors generated by one or more color sources incombination is maximized.
 8. A method of allocating display timescomprising: receiving image data for an image to be displayed during aframe time using one or more primary colors; converting the image datato a sequence of bit planes for a set of colors including the primarycolors and secondary colors; calculating a recycling gain for each bitplane; compressing each bit plane based on the recycling gain;determining an unused portion of the frame time based on the compressedbit planes; and stretching the bit planes to utilize the unused portionof the frame time.
 9. The method of claim 8, comprising: displaying thestretched bit planes.
 10. The method of claim 8, wherein converting theimage data increases the contribution of the secondary colors.
 11. Themethod of claim 8, wherein converting the image data maximizes thecontribution of the secondary colors.